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ABSTRACT 

This paper reports the partitioning of varianee in seale seores from the use of three 
elassroom environment instruments. Data sets from the administration of the What Is 
Happening In this Class (WIHIC) to 4,146 students, the Questionnaire on Teaeher 
Interaetion (QTI) to 2,167 students and the Catholie Sehool Classroom Environment 
Questionnaire (CSCEQ) to 2,21 1 students in Australian sehools were analysed using 
MLwiN. Varianee eomponents models with three levels (viz. student, elass and 
sehool) for eaeh of the 22 seales were ereated. Results revealed that statistieally 
signifleant proportions of varianee in all seale seores eould be attributed to the 
student and elass levels. Apart from one seale, no sehool level varianee in seale 
seores was statistieally signifleant. Typieally, varianee proportions were above 75% 
at the student level, above 15% at the elass level, and below 5% at the sehool level. 
Multilevel analyses should be eondueted with elassroom environment data. 


INTRODUCTION 

Research on the psychosocial dimensions of classroom environments has made 
substantial progress during the past 40 years. This research has focused mainly on the 
individual student’s psychological development and how this interacts with the social 
environment. Accordingly the concept of environment, as applied to educational settings, 
refers to the atmosphere, ambience, tone, or climate that pervades the particular setting. The 
strong methodological tradition of classroom environment research has been to conceptualise 
environments in terms of Murray’s (1938) beta press - the perceptions of the milieu 
inhabitants (i.e. students and teachers) - with context-specific instruments assessing particular 
dimensions of the learning environment. Axiomatic with this methodological tradition has 
been the long standing “unit of analysis problem’’ (Burstein, Fischer, & Miller, 1980) which 
draws attention to the issue of how to analyse data from students in schools. More 
specifically, researchers who use instruments to study classroom environment invariably 
survey students within classes within schools. Because beta press assesses classroom 
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environment from the student’s perspective, the nested or hierarchical nature to the data is 
inescapable. A central issue in the analysis of such data concerns the distribution or 
partitioning of variance across the levels of the hierarchy. How is variance in student scores 
distributed across the levels? For students in schools, this will most likely involve three 
levels: student, class and school. The purpose of the present paper is to draw attention to this 
issue by partitioning variance on scale scores from the use of three classroom environment 
instruments. Before describing the research, its results and implications, the following section 
provides background information on classroom environment research and variance 
partitioning. 

Classroom Environment Research 

Reviews of classroom environment research by Fraser (2007) and Dorman (2002) and 
edited books by Rhine and Fisher (2003) and Fisher and Rhine (2006) have reported research 
on the evaluation, antecedents and consequences of classroom environments. These studies 
have included associations between classroom environment and outcomes, evaluation of 
educational innovations, differences between students’ and teachers’ perceptions of 
classrooms, comparisons of actual and preferred environments, effect of determinants or 
antecedents on classroom environment of antecedent variables (e.g. student gender, year, 
subject, school type), transition from primary to secondary school, school psychology, student 
meta-cognition, teacher education, educational productivity research, and using environment 
instruments to facilitate changes in classroom life. 

The use of the students’ perceptions to assess classroom environment can be linked 
conceptually to Lewin’s (1936) Field Theory and the Lewinian formula B =/(P, E) (i.e. 
Behaviour is a function of person and the environment as it exists for that person.). Murray’s 
(1938) need-press theory posited the notion that individuals have internal needs and the 
external environment or press either supports or frustrates that need. Since the late 1960s, an 
overwhelming feature of classroom environment research and associated instrumentation has 
been the collection and analysis of high- inference student and teacher perceptual data. Such 
high-inference instruments require respondents to make judgments based on a sustained 
period of time in the classroom using specific constructs (e.g. involvement). By contrast, low- 
inference measures focus on discrete classroom phenomena and usually involve frequency 
counts (e.g. How many times did the teacher say “ok” in the 90 minute lesson?) Studies 
which focus on the meaning of school and classroom events have tended to utilize high- 
inference measures as advocated strongly by Walberg (1976). That is, students should be 
asked to make summary molar judgments about their classrooms rather than piecemeal 
reporting on a myriad of molecular events. 

Over the past 40 years, a suite of classroom environment instruments have been 
developed, validated and used in a host of school settings. Some of these instruments include 
the Classroom Environment Scale (CES: Moos & Trickett, 1987), the Learning Environment 
Inventory (EEI: Fraser, Anderson, & Walberg, 1982), the Individualized Classroom 
Environment Questionnaire (ICEQ: Fraser, 1990), the Questionnaire on Teacher Interaction 
(QTI (Wubbels & Eevy, 1993) and the What Is Happening In this Class (WIHIC: Fraser, 
2007). Additionally, a plethora of context-specific derivatives of these instruments have been 
used in research in specific settings, particularly by doctoral students. The number of 
instrument derivatives is high and, as Shavelson and Seidel (2006) remarked, there is a need 
for the learning environment community to engage in an instrument culling program. 

It is also important to note the methodological advances in the development, validation 
and use of classroom environment instruments over the past decade. At the instrument 
development level, Cavanagh and Romanoski (2006) set out a case for Rasch modelling in the 
development of true measures of learning environments. This approach was illustrated with 
the development of a classroom learning culture scale. 

Confirmatory factor analysis has been employed in recent studies. Dorman (2003) 
employed EISREE to establish the factorial invariance of the WIHIC according to country, 
gender and year of student. Den Brok, Fisher, Wubbels, Brekelmans, and Rickards (2006) 
performed multigroup confirmatory factor analysis on QTI data collected in Singapore, 
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Brunei and Australia. Dorman (in press) performed a multitrait-multimethod analysis of 
actual and preferred versions of the WIHIC. Structural equation modelling and multilevel 
modelling have advanced the data analysis techniques employed in the learning environment 
field (see den Brok, Brekelmans, & Wubbels, 2006; den Brok, Fisher, Rickards, & Bull, 

2006; Dorman, Fisher, & Waldrip, 2006; Fisher, Waldrip, & den Brok, 2005). 

Some areas of contemporary classroom environment research include monitoring 
interpersonal behavior and student outcomes in vocational classes (Flenderson & Fisher, 

2008), investigating parent and student perceptions of classroom environments (Allen & 
Fraser, 2007), studying the classroom climate and students’ goal structures in secondary 
school biology classes in Kenya (Mucherah, 2008), and investigating the effect of extended 
instructional time on learning environment, achievement, and attitudes in middle schools 
algebra classes (Azimioara & Fraser, 2007). Recent edited volumes have documented the 
growth in learning environment research over the past decade (see Fisher & Rhine, 2006; Goh 
& Rhine, 2002; Rhine & Fisher, 2003). Aldridge and Fraser’s (2008) recent book describes 
the learning environment of a school with an outcomes-based focus. Classroom environment 
literature has significant overlap with aspects of the student engagement literature (see 
Fredricks, Blumenfeld, & Paris, 2004). Establishing cohesive classroom environments that 
have high quality relationships and active student participation are critical to student 
academic engagement. 

Variance Partitioning 

There are many research settings that involve nested data. That is, a data hierarchy exists 
in which lower level units exist within higher level units. In hospitals, patients might be 
nested in wards which are nested within specialist units which, in turn, are located in 
hospitals. Voters are located in electorates which could be located in states. The most obvious 
hierarchy in educational settings is a three level school hierarchy: students are nested within 
classes that are nested within schools. Students within a class experience the same classroom 
learning environment. In turn, classes within a school experience a different overall school 
environment compared to classes in other schools. Acceptance of this hierarchy suggests that 
students’ responses to questionnaires will not reflect the idiosyncratic views of students alone. 
While two students in the same class may respond to a classroom environment questionnaire 
in different ways, the common classroom environment will have some influence on their 
perceptions. That is, group membership effects exist. 

A fundamental issue concerning group effects is that even if individuals are assigned to 
groups on a random basis, they, as a group will become differentiated. Students influence, and 
are influenced by other students in the class (Goldstein 2003). It is also true that schools can 
create class effects by directing students to classes on biased bases (e.g., timetabling 
constraints, specialist teacher availability, subject choice, specialist classroom availability). In 
essence, variance in students’ scores can be partitioned at the student, class and school levels. 
The intraclass correlation, p or variance partition coefficient (VPC) is the proportion of 
variance accounted for by higher level units and can be thought of as the “extent of 
clustering” (Goldstein, Browne, & Rasbash, 2002). Qualitatively, the VPC can be taken as a 
measure of the importance of the particular level. So the computed value for the VPC for 
classes provides an indication of how important class membership is to scores on the 
particular variable under consideration. 

While the hierarchical/nested nature of clustered data is clear, this essential characteristic 
has often been ignored when analysing data. Analysts have used either the individual as the 
unit of analysis and ignored class membership or the class as the unit of analysis with 
aggregated data and thus ignored the individual student. In response to these criticisms, some 
researchers have reported parallel but essentially independent sets of analyses conducted with 
both the individual student and the class as units of analysis in the one study (e.g. Goh & 
Fraser 1998). 

According to Snijders and Bosker (1993), intraclass correlations for most educational 
settings range typically from .05 to .20. However, parameter values are dependent on the 
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setting and variables under investigation. Lee (2000) asserted that a variance proportion above 
10% at any level is non-trivial and needs to be taken into account in any analysis. Roberts 
(2007) was particularly critical of intraclass correlation thresholds as precursors to multilevel 
analysis. He cautioned that, even with intraclass correlations near zero, group dependence can 
exist when variables are added to the model. 

One key assumption of statistical tests like analysis of variance and t tests is that subjects 
are statistically independent. This assumption is violated if non-trivial intraclass correlations 
exist. Dorman (2008a, 2008c) has demonstrated the dramatic effect on Type I error rates if 
clustering is ignored when conducting such statistical tests. For example, in a two-group 
comparison involving 20 classes per group and 25 students per class, a nominal t test Type 1 
error rate of .05 is inflated to an actual Type I error rate of .418 if the intraclass correlation is 
.20. That is, the probability of rejecting the null hypothesis and concluding that statistically 
significant differences exist between the groups when, in fact they do not, has inflated over 8 
times to a clearly unacceptable value. Murray, Hannan and Baker (1996) noted that 
investigators who employ an analysis at the level of the individual run a very real risk of 
overstating the statistical significance reported for the test. 

Proponents of multilevel modelling have argued that the existence of grouping hierarchies 
in data is neither accidental nor ignorable (Rowe, 2007) and that data with a clear hierarchy 
should not be analysed as if they are all on the same level because it can lead to statistical and 
interpretational errors (Tabachnick & Fidell, 2007). Dorman (2008b) recently demonstrated 
the merits of multilevel analysis in learning environment research by comparing the results of 
analyses conducted with the individual and class as units of analysis with the results of 
multilevel analysis. Classroom environment research of the type described in the above 
section inevitably involves nested data. The hierarchical nature to these data is self-evident 
and the partitioning of variance across the levels in the hierarchy is a significant first step in 
the conduct of multilevel analyses. 

The aim of this research was to investigate the extent to which scores on three classroom 
environment instruments assess aspects of the classroom environment as opposed to either the 
idiosyncratic assessments of individual students or the overall school environment. This aim 
requires the partitioning of variance in scales scores across three levels: the individual 
student, the class and the school. As described below, the present investigation analysed data 
from three separate Australian classroom environment studies which employed the What Is 
Happening In this Class (WIHIC) questionnaire, the Questionnaire on Teacher Interaction 
(QTI) and the Catholic School Classroom Environment Questionnaire (CSCEQ). 

METHOD 


Samples 

Table 1 describes the samples in terms of student gender and year. These students were 
from primary and secondary schools mainly in Queensland and Western Australia. The 
respective class and school membership for these students were: WIHIC, 4,146 students from 
286 classes in 21 schools; QTI, 2,167 students from 103 classes in 37 schools , and CSCEQ, 
2,21 1 students from 104 classes in 32 schools. 

Instrumentation 

Three different instruments were used in the three independent studies described in this paper. 
The first instrument is the What Is Happening In this Class? (WIHIC) questionnaire which is 
a well-established and widely-used instrument in classroom environment research (see, e.g. 
Dorman, 2003). The WIHIC scales are: student cohesiveness, teacher support, involvement, 
investigation, task orientation, cooperation and equity. The WIHIC consists of 56 items 
assigned to 7 underlying scales (8 items per scale). Each item employs a 5-point Eikert 
response format (viz. almost never = 1, seldom = 2, sometimes = 3, often = 4, almost always 
= 5) with items scores aggregated to form scale scores for each respondent. Table 2 shows 
descriptions of each WIHIC scale. 
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Table 1: Description of the three samples 

in this study. 



Year 



Sample Size 



WIHIC 

QTI 

CSCEQ 

Male 

Female 

Male 

Female 

Male 

Female 

5-7 

- 

- 

1051 

1112 

- 

- 

8-10 

206 

247 

- 

- 

584 

598 

11-12 

1729 

1964 

- 

- 

489 

540 

Total 

1935 

2211 

1051 

1112 

1073 

1138 


The robust nature of the WIHIC’s reliability and validity has been widely reported in 
studies that have used the instrument in different subject areas, at different age levels and in 
12 different countries. Since the initial development of the WIHIC, the questionnaire has been 
used successfully in studies to assess the learning environment in Australia and Taiwan 
(Aldridge, Fraser, & Fluang, 1 999), Canada (Zandvliet & Fraser, 2004), Australia (Dorman, 

2001) , Turkey (Telli, Cakiroglu, & den Brok, 2006), New Zealand (Saunders & Fisher, 2006), 
the United States (Allen & Fraser, 2007), and Canada, England and Australia (Dorman, 

2003). Within these countries, the WIFIIC has assessed the environment in a range of 
curriculum areas including secondary school science (Aldridge et ah, 1999), mathematics 
(Margianti, Fraser, & Aldridge, 2004) and mathematics and science (Raaflaub & Fraser, 

2002) . 

The second instrument, the Questionnaire on Teacher Interaction (QTI: Wubbels & Levy, 
1993) was developed in The Netherlands to focus on one aspect of classroom environment: 
teacher communication style. Its conceptual basis is a model of interpersonal teacher 
behaviour which was developed from Leary’s (1957) personality model of interpersonal 
behaviour. According to den Brok, Fisher, Wubbels, et al. (2006) various forms of the QTI 
have been employed in over 120 studies worldwide. For example, Waldrip, Reene, Fisher, 
and Dorman (2008) used the QTI to assist teachers to make changes to their teaching 
strategies. QTI scores are used to map teacher behaviour on two orthogonal dimensions, a 
proximity dimension (cooperation-opposition) and an influence dimension (dominance- 
submission). This leads to eight different behaviour types with eight corresponding scales: 
leadership, helpful/friendly, understanding, student responsibility/freedom, uncertain, 
dissatisfied, admonishing, and strict (see Table 2). The version of the QTI used in the present 
study has 48 items (6 items per scale) and each item used a 4-point response format with 
anchors of agree and disagree. 

The third instrument is the Catholic School Classroom Environment Questionnaire 
(CSCEQ) which consists of 66 items assigned to 7 underlying scales: student affiliation, 
interactions, cooperation, task orientation, order and organization, individualization, and 
teacher control (see Table 2). It is noteworthy that the CSCEQ is an amalgam of existing 
scales and newly developed scales to assess catholic school classroom environments. 

Whereas the task orientation and teacher control scales are from Moos and Tricketf s (1987) 
Classroom Environment Scale, the interactions scale was developed by Dorman (1994). 
CSCEQ scales have either 9 or 10 items. Each item employs a 5-point Likert response format 
(viz. strongly disagree = 1, disagree = 2, not sure = 3, agree = 4, strongly agree = 5) with 
items scores aggregated to form scale scores for each respondent. Different versions of the 
CSCEQ have been used in previous research. For example, Dorman, McRobbie, and Foster 
(2002) used a personalized form of the CSCEQ to study associations between classroom 
environment in religion classes and their attitudes to Christianity. The CSCEQ has been 
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shown to provide a valid assessment of classroom environment, especially in Australian 
Catholic schools. 

Table 2: Descriptive information for scales from three classroom environment instruments. 


Seale 

Seale Deseription 

Cronbach 

a 

M 

SD 

WIHIC 

The extent to which: 




Student 

Cohesiveness 

students know, help and are supportive of one another. 

0.89 

4.55 

0.80 

Teaeher Support 

the teaeher helps, befriends, trusts and is interested in 
students. 

0.93 

3.99 

0.98 

Involvement 

students have attentive interest, partieipate in diseussions, do 
additional work and enjoy the elass. 

0.91 

3.72 

0.93 

Task Orientation 

it is important to eomplete aetivities planned and to stay on 
the subjeet matter. 

0.89 

4.53 

0.81 

Investigation 

skills and proeesses of inquiry and their use in problem 
solving and investigation are emphasised. 

0.94 

3.32 

1.02 

Cooperation 

students eooperate rather than eompete with one another on 
learning tasks. 

0.92 

4.26 

0.88 

Equity 

students are treated equally by the teaeher. 

0.95 

4.39 

0.91 

QTI 

The extent to which the teacher: 




Leadership 

notiees what is happening, leads, organizes, gives orders, 
sets tasks and determines elassroom proeedures. 

0.67 

3.06 

0.56 

Helpful/friendly 

assists, shows interest, behaves in a friendly or eonsiderate 
manner and inspires eonfidenee and trust. 

0.81 

3.24 

0.65 

Understanding 

listens with interest, sympathizes, shows eonfidenee and 
understanding, aeeepts apologies and is patient. 

0.80 

3.12 

0.63 

Student 

gives opportunity for independent work, waits for elass to let 

0.72 

2.74 

0.58 

responsibility/ 

freedom 

off steam, gives freedom and responsibility. 




Uneertain 

keeps a low profile, apologizes, waits and sees how the wind 
blows, and admit one is wrong. 

0.62 

1.85 

0.56 

Dissatisfied 

waits for silenee, eonsiders pros and eons, keeps quiet, 
shows dissatisfaetion, looks glum and eritieizes. 

0.72 

1.84 

0.63 

Admonishing 

gets angry, takes pupils to task, express irritation, and anger, 
forbids, eorreets and punishes. 

0.77 

2.04 

0.70 

Striet 

keeps a tight rein, eheeks, gets elass silent, maintains 
silenee, is striet and has exaet norms and rules. 

0.65 

2.42 

0.61 

CSCEQ 

The extent to which: 




Student 

Affiliation 

students know, help and are friendly towards eaeh other. 

0.69 

3.49 

0.54 

Interaetions 

teaeher-student interaetions emphasize a eoneem for the 
personal welfare and soeial growth of the student. 

0.90 

3.76 

0.74 

Cooperation 

students eooperate rather than eompete with eaeh other. 

0.71 

3.22 

0.53 

Task Orientation 

Extent to whieh it is important to eomplete aetivities planned 
and to stay on the subjeet matter. 

0.76 

3.25 

0.65 

Order & 

students behaving in an orderly, quiet and polite manner. 

0.84 

2.75 

0.69 

Organization 

and on the overall organization of elassroom aetivities. 

Individualisation 

students are allowed to make deeisions and are treated 
differently aeeording to ability, interest and rate of working. 

0.54 

2.64 

0.46 

Teaeher Control 

rules are enforeed and how severely infraetions are 
punished. 

0.75 

3.47 

0.60 
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Data Analysis 

All three data sets used in this investigation of variance at different levels involved 
students nested within classes within schools. That is, the data are hierarchical at three levels. 
As indicated above, these three instruments have been employed in substantial previous 
research and have very sound structural characteristics. Accordingly, confirmatory factor 
analyses are not reported here. However, factor score regressions derived from these factor 
analyses were used to weight items when computing scale scores. This approach minimizes 
measurement error variance for each scale (see Holmes-Smith & Rowe, 1994). Scores for all 
22 scales employed in this study were normalized prior to regression analyses. This approach 
attenuated the effect of non-normal univariate and multivariate scale score distributions, 
especially with regard to departures from normality in scale kurtosis. These normal scores 
were employed in all analyses reported below. Three-level multilevel analyses were 
conducted using MLwiN (Rasbash, Steele, Browne, & Prosser, 2005) with a base variance 
components (i.e. null) model created for each scale. 


RESULTS 


Table 2 shows the Cronbach coefficient a for each of the 22 scales used in this 
investigation. These values range from 0.54 for the investigation scale of the CSCEQ to 0.95 
for the equity scale of the WIHIC. Overall the scales have very sound internal consistency 
reliability. As a Cronbach coefficient a of .54 is only marginally acceptable, results for the 
investigation scale need to be treated with caution. Table 2 also reports means and standard 
deviations for each scale. 

Results of the variance components models are shown in Table 3. The most striking 
observation from these results is that all between student and between class variances were 
statistically significant (p<.05). Apart from the cooperation scale of the CSCEQ, all school 
level variances were not statistically significant. As expected, most of the variance in scales 
scores was at the student level with the proportion of variance at this level ranging from the 
QTTs admonishing (69.40%) to the WIHIC’s task orientation (92.52%). However, it is 
noteworthy that 8 of the 22 scales had infra class correlations (the proportion of variance at 
the class level) over 20%. The WIHIC’s task orientation had the lowest infra class correlation 
(6.18%). The highest intraschool school correlation (the proportion of variance at the school 
level) was for the QTTs strict scale (7.89%). It is also noteworthy that the average infra class 
correlation was 12.59% for the WIHIC, 18.55% for the QTI and 19.08% for the CSCEQ. 

DISCUSSION 

This discussion falls into two areas. Issues arising from this research concerning the 
importance of multilevel analysis in classroom environment research are discussed. This 
discussion is followed by a comparison of the present study’s results with the findings of 
previous similar classroom environments studies. 

The important methodological finding of this paper is that the multilevel nature to 
classroom environment data is real and needs to be acknowledged in approaches to data 
analysis. There are very few classroom environment studies in which data are not collected 
from intact classes. According to Goldstein (2003), despite the fact that classes are often 
established on a random basis, they tend to become differentiated and that students influence 
and are influenced by other students in the class. That is, a class effect comes into being. It is 
also true that schools can create class effects by directing students to classes on a clearly 
biased basis (e.g. timetabling, specialist teacher availability, subject choice). If such class 
effects are ignored, statistical inferences may be invalid. As noted by Hox (1995), analysis of 
variance may overestimate the effects of variables at the student level if group effects are not 
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Table 3: Parameter estimates and variance components for null models for three classroom environment instruments. 



Fixed 



Random (Residual Variance) 



Scale 


Between Students 

Between Classes 

Between Schools 


School Intercept 


% 


% 


% 

WIHIC 








Student Cohesiveness 

-0.107 (0.077) 

0.843 (0.020)* 

88.73 

0.062 (0.011)* 

6.53 

0.045 (0.026) 

4.74 

Teacher Support 

0.043 (0.059) 

0.725 (0.017)* 

79.32 

0.173 (0.020)* 

18.93 

0.016(0.014) 

1.75 

Involvement 

-0.089 (0.080) 

0.845 (0.020)* 

87.56 

0.070 (0.012)* 

7.26 

0.050 (0.028) 

5.18 

Task Orientation 

0.010 (0.048) 

0.853 (0.020)* 

92.52 

0.057(0.010)* 

6.18 

0.012(0.010) 

1.30 

Investigation 

-0.027 (0.056) 

0.878 (0.021)* 

91.55 

0.063 (0.011)* 

6.57 

0.018(0.013) 

1.88 

Cooperation 

-0.090 (0.072) 

0.843 (0.020)* 

87.90 

0.079 (0.013)* 

8.24 

0.037 (0.023) 

3.86 

Equity 

0.002 (0.034) 

0.755 (0.018)* 

87.18 

0.109 (0.015)* 

12.59 

0.002 (0.004) 

0.23 

QTI 

Leadership 

0.008 (0.046) 

0.823 (0.026)* 

84.85 

0.138 (0.029)* 

14.22 

0.009 (0.017) 

0.93 

HelpfuEfriendly 

-0.007 (0.050) 

0.716(0.022)* 

78.68 

0.186 (0.036)* 

20.44 

0.008 (0.019) 

0.88 

Understanding 

-0.001 (0.049) 

0.752 (0.024)* 

80.17 

0.178 (0.035)* 

18.98 

0.008 (0.019) 

0.85 

Student responsibility/ 
freedom 

0.004 (0.051) 

0.780 (0.024)* 

80.33 

0.181 (0.035)* 

18.64 

0.010 (0.020) 

1.03 

Uncertain 

-0.020 (0.054) 

0.797 (0.025) 

83.55 

0.121 (0.027) 

12.68 

0.036 (0.024) 

3.77 

Dissatisfied 

0.005 (0.058) 

0.698 (0.022) 

75.38 

0.199 (0.038) 

21.49 

0.029 (0.028) 

3.13 

Admonishing 

0.012 (0.068) 

0.651 (0.020) 

69.40 

0.230 (0.044) 

24.52 

0.057 (0.039) 

6.08 

Strict 

-0.013 (0.069) 

0.729 (0.023) 

74.69 

0.170 (0.035) 

17.42 

0.077 (0.040) 

7.89 

CSCEQ 








Student Affiliation 

0.030 (0.054) 

0.830 (0.026)* 

82.84 

0.134 (0.029)* 

13.37 

0.038 (0.025) 

3.79 

Interactions 

0.028 (0.069) 

0.722 (0.022)* 

72.42 

0.201 (0.039)* 

20.16 

0.074 (0.039) 

7.42 

Cooperation 

0.013 (0.054) 

0.862 (0.027)* 

86.11 

0.090 (0.022) * 

8.99 

0.049 (0.024) * 

4.90 

Task Orientation 

-0.002 (0.053) 

0.740 (0.023)* 

74.22 

0.257 (0.041)* 

25.78 

- 

- 

Order & Organization 

0.030 (0.067) 

0.718(0.022)* 

71.51 

0.229 (0.044) * 

22.81 

0.057 (0.037) 

5.68 

Indi viduali sation 

0.030 (0.052) 

0.781 (0.024)* 

78.41 

0.202 (0.039)* 

20.28 

0.013 (0.024) 

1.31 

Teacher Control 

-0.018 (0.050) 

0.775 (0.024)* 

77.81 

0.221 (0.036)* 

22.19 

- 

- 


*/? <.05 
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taken into account. If, on the other hand, student data are aggregated to the class level, 
information and statistical power is lost. Additionally, the potential for making ecological 
fallacies in which inferences about students are based on data analyses performed at the class 
level is very real (see Alker, 1969). 

In the present study, little variance in some scale scores was evident at the school level. 
Depending on the research questions it would not be unreasonable to remove the school level 
from some models in which classroom environment scales are explanatory or response 
variables. However, it is advisable to check for variance at all levels of the hierarchy before 
conducting subsequent analyses in a particular study. As such, multilevel analyses should 
start with variance components models for all explanatory and response variables with all 
appropriate levels before fitting conditional models. It should be noted that when school is 
removed in the present three level variance components models, all the between school 
variance is shifted to the between class variance. In general, ignoring or removing a level 
affects parameter estimates, standard errors and variance partitioning of the flanking levels 
(i.e. those levels above and below the ignored level) (see van den Noortgate, Opdenakker, & 
Onghena, 2005). 

The magnitude of the effect of ignoring or removing a level will depend on the proportion 
of variance that would have been accounted for by the ignored level. For example in the 
present study, if school is removed as the third level variable in variance partitioning models 
for the CSCEQ’s task orientation and teacher control scales, there would be no change in the 
proportion of variance in scale scores at the class level as there was no variance accounted for 
at the school level for these two scales (see Table 3). However, if school is removed as the 
third level variable in variance partitioning models for the QTTs strict scale, the shift in class 
level variance is from 17.42% to 25.31% - a sizeable increase. The effects on variance 
partitioning due to ignoring levels in variance components models can be appreciable (van 
den Noortgate, Opdenakker, & Onghena, 2005). 

The second part of this discussion attempts to link the findings of the present study with 
previous classroom environment research. As noted earlier in this paper, despite the obvious 
nested nature to the data in almost all classroom environment studies, only a handful of 
reported studies have employed multilevel analysis. Nevertheless, several recent studies have 
used multilevel analysis and it is important to compare the findings of the present study of 
variance partitioning with these studies. It is noteworthy that some studies have modelled data 
at two levels (student and class), others at three levels (student, class and school) and one 
study at four levels (student, class, teacher and school). 

In the only study to date that has reported variance partitioning of the WIHIC using 
multilevel analysis, den Brok, Fisher, Rickards, and Bull (2006) collected data from 655 
students in 28 science classes in 1 1 Californian schools. Variance partitioning with a three- 
level model indicated that most of the variance was at the student level with some variance at 
the class level and only a very small amount at the school level. For example, variance in 
student cohesiveness was distributed as student, 98.60%; class, 1.12%; and school 0.28%. 
Overall, den Brok et al.’s results are very similar to the WIHIC results of the present study 
with teacher support and equity having appreciable amounts of variance at the class level 
(16.05% and 23.86% respectively). In the present study these same two scales had the largest 
proportions of class level variance: teacher support, 18.93% and equity, 12.59% (see Table 3). 
One cautionary note identified by den Brok et al. is the relatively small number of classes and 
schools in their sample. The present study employed a large sample of classes (286) but in a 
relatively small number of schools (21). The results of den Brok et al.’s study and the present 
study are consistent in that variances in WIHIC scale scores are overwhelmingly at the 
student level. 

Two studies involving variance partitioning of scores on the Questionnaire on Teacher 
Interaction (QTI) were conducted by Levy, den Brok, Wubbels, and Brekelmans (2003) and 
den Brok, Brekelmans, and Wubbels (2006). In the present study, while most of the variance 
was at the student level, there were appreciable proportions of variance (ranging from 12.68% 
for uncertain to 24.52% for admonishing) for all eight QTI scales at the class level. In the 
Levy et al. study, variance was apportioned to four levels: student, class, teacher and school. 
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Student-level variance ranged from 69% for leadership to 77.3% for uncertain with class and 
teacher factors each accounting for 10-15%. There is a high level of consistency between the 
finding of Levy et al.’s study and the present study. 

The study by den Brok, Brekelmans, and Wubbels (2006) involved data from both class 
and personal versions of the QTI collected in The Netherlands. Variances in QTI scale scores 
were partitioned at the student and class levels. Intra class correlations for the class and 
personal version were very similar and ranged from 0.23 for the personal form of the strict 
scaled to 0.5 1 for personal form of the leadership scale. All intra class correlations indicated 
that sizeable amounts of variance in scores were at the class level. There was significant 
consensus within classes on several scales, especially leadership, helpfiil/friendly, 
understanding, uncertain, dissatisfied and admonishing for both class and personal versions. 

While no previous variance partitioning has been reported for the CSCEQ, it is instructive 
to consider other classroom environment studies in which variances at different levels have 
been reported. Fisher et al. (2005) reported the use of the equity, collaboration, and 
congruence scales of the Cultural Learning Environment Questionnaire (CLEQ) with 2,178 
Australian primary school students in 103 classrooms. For these scales, over 90% of the 
variance in scores was at the student level. Interestingly, for the congruence scale, which 
refers to the level of similarity between the home and school environments, a greater 
proportion of variance was attributed to the school level (6.4%) compared to the class level 
( 2 . 1 %). 

As three of the congruence scale’s five items referred to the school and not the classroom, 
this result is entirely plausible. 

Wheldall, Beaman and Mok (1999) used Fraser’s (1990) Individualized Classroom 
Environment Questionnaire (ICEQ) in a study of variance partitioning at the student, class 
and school levels. While the sample involved 1,467 students in 81 classes, they were drawn 
from only four secondary schools in Sydney, Australia. Accordingly there must be a degree of 
uncertainty with regard to estimated variances and coefficients. Models with a small number 
of observations in higher levels typically lack statistical power. This study revealed 
significant proportions of variance at the class level for all five scales. Intra class correlations 
were above 1 6% for the personalization, participation, independence and differentiation 
scales. The intra school correlations ranged from 1.5% for the participation scale to 13.1% for 
the personalization scale. 

Two final studies highlight previous research in which variance partitioning could have 
been reported. Goh and Fraser (1998) reported multilevel analyses of QTI and My Class 
Inventory (Fraser et al., 1982) scores and outcome scales. However they did not report 
variance partitioning from a multilevel perspective. Although eta^ (the proportion of variance 
explained by class membership) was reported for each scale via individual one-way analysis 
of variance for class membership, this approach does not reflect the multilevel nature of the 
data. Similarly, Wong, Young, and Fraser (1997) did not report variance partitioning of scale 
scores in their multilevel study of chemistry classroom environment in Singapore. 

Overall, the results of the present study are consistent with previous research on variance 
partitioning of scores on classroom environment scales. One noteworthy feature of the present 
results is that the proportions of variances in QTI scores at the class level are consistently 
high, especially when compared to the WIHIC’s results and other research findings discussed 
above. As the QTI focuses explicitly on teacher interaction with students, these findings are 
not illogical. It is plausible that students in a class would hold a more collective view of the 
teacher’s classroom interactions than say, personal growth dimensions. For example the 
cooperation scales of the WIHIC and CSCEQ probably assess students’ idiosyncratic views 
rather than an overall classroom viewpoint. 

One noteworthy issue concerning the results in the present study concerns the school 
types where the data were collected. The QTI data were from primary school students and the 
WIHIC and CSCEQ data were from secondary school students (see Table 1). Whereas 
students in primary schools largely experience one classroom environment, students in 
secondary schools experience several classroom environments. Accordingly, the proportions 
of variance at the classroom level with the QTI scores could be related to these students being 
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from primary schools compared to the other two studies which were conducted in secondary 
schools. 


CONCLUSION 

The present study assessed variance partitioning of scores on three classroom 
environment instruments: the What Is Happening In this Class (WIHIC), the Questionnaire on 
Teacher Interaction (QTI) and the Catholic School Classroom Environment Questionnaire 
(CSCEQ). Multilevel variance components models (i.e. null models) with three levels (viz. 
student, class and school) were used to separate variance into these three levels. By taking 
into account the hierarchical nature of these data, this study has shown that the class level 
accounted for significant amounts of variance in scores on all scales of all three instruments. 
As noted above, the proportion of variance at the class level varied considerably among these 
22 scales. This suggests that, whereas some scales are more aligned with a summary 
classroom atmosphere, other scales assess the distinctive views of individual students within 
classes. It is an inevitable fact that almost all classroom environment data are nested. Students 
are taught in classes and classes remain the basic instructional unit of schools. The important 
conclusion from this study is that this multilevel nature to classroom environment data cannot 
be ignored. There is simply too much variance in scales scores at the class level to use the 
individual as the unit of analysis. However the potential for aggregation bias and ecological 
fallacies makes the use of the class as the unit of analysis problematic also. Neither using the 
individual nor the class as units of analysis is optimal and this paper recommends the use of 
multilevel modelling when analysing classroom environment data. 
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